26 research outputs found
One-Shot Labeling for Automatic Relevance Estimation
Dealing with unjudged documents ("holes") in relevance assessments is a
perennial problem when evaluating search systems with offline experiments.
Holes can reduce the apparent effectiveness of retrieval systems during
evaluation and introduce biases in models trained with incomplete data. In this
work, we explore whether large language models can help us fill such holes to
improve offline evaluations. We examine an extreme, albeit common, evaluation
setting wherein only a single known relevant document per query is available
for evaluation. We then explore various approaches for predicting the relevance
of unjudged documents with respect to a query and the known relevant document,
including nearest neighbor, supervised, and prompting techniques. We find that
although the predictions of these One-Shot Labelers (1SL) frequently disagree
with human assessments, the labels they produce yield a far more reliable
ranking of systems than the single labels do alone. Specifically, the strongest
approaches can consistently reach system ranking correlations of over 0.86 with
the full rankings over a variety of measures. Meanwhile, the approach
substantially increases the reliability of t-tests due to filling holes in
relevance assessments, giving researchers more confidence in results they find
to be significant. Alongside this work, we release an easy-to-use software
package to enable the use of 1SL for evaluation of other ad-hoc collections or
systems.Comment: SIGIR 202
RSDD-Time: Temporal Annotation of Self-Reported Mental Health Diagnoses
Self-reported diagnosis statements have been widely employed in studying
language related to mental health in social media. However, existing research
has largely ignored the temporality of mental health diagnoses. In this work,
we introduce RSDD-Time: a new dataset of 598 manually annotated self-reported
depression diagnosis posts from Reddit that include temporal information about
the diagnosis. Annotations include whether a mental health condition is present
and how recently the diagnosis happened. Furthermore, we include exact temporal
spans that relate to the date of diagnosis. This information is valuable for
various computational methods to examine mental health through social media
because one's mental health state is not static. We also test several baseline
classification and extraction approaches, which suggest that extracting
temporal information from self-reported diagnosis statements is challenging.Comment: 6 pages, accepted for publication at the CLPsych workshop at
NAACL-HLT 201
Characterizing Question Facets for Complex Answer Retrieval
Complex answer retrieval (CAR) is the process of retrieving answers to
questions that have multifaceted or nuanced answers. In this work, we present
two novel approaches for CAR based on the observation that question facets can
vary in utility: from structural (facets that can apply to many similar topics,
such as 'History') to topical (facets that are specific to the question's
topic, such as the 'Westward expansion' of the United States). We first explore
a way to incorporate facet utility into ranking models during query term score
combination. We then explore a general approach to reform the structure of
ranking models to aid in learning of facet utility in the query-document term
matching phase. When we use our techniques with a leading neural ranker on the
TREC CAR dataset, our methods rank first in the 2017 TREC CAR benchmark, and
yield up to 26% higher performance than the next best method.Comment: 4 pages; SIGIR 2018 Short Pape
SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions
Mental health is a significant and growing public health concern. As language
usage can be leveraged to obtain crucial insights into mental health
conditions, there is a need for large-scale, labeled, mental health-related
datasets of users who have been diagnosed with one or more of such conditions.
In this paper, we investigate the creation of high-precision patterns to
identify self-reported diagnoses of nine different mental health conditions,
and obtain high-quality labeled data without the need for manual labelling. We
introduce the SMHD (Self-reported Mental Health Diagnoses) dataset and make it
available. SMHD is a novel large dataset of social media posts from users with
one or multiple mental health conditions along with matched control users. We
examine distinctions in users' language, as measured by linguistic and
psychological variables. We further explore text classification methods to
identify individuals with mental conditions through their language.Comment: COLING 201
SMHD : a large-scale resource for exploring online language usage for multiple mental health conditions
Mental health is a significant and growing public health concern. As language usage can be leveraged to obtain crucial insights into mental health conditions, there is a need for large-scale, labeled, mental health-related datasets of users who have been diagnosed with one or more of such conditions. In this paper, we investigate the creation of high-precision patterns to identify self-reported diagnoses of nine different mental health conditions, and obtain high-quality labeled
data without the need for manual labelling. We introduce the SMHD (Self-reported Mental Health Diagnoses) dataset and make it available. SMHD is a novel large dataset of social media posts from users with one or multiple mental health conditions along with matched control users. We examine distinctions in users’ language, as measured by linguistic and psychological variables. We further explore text classification methods to identify individuals with mental conditions
through their language
The Surveillance AI Pipeline
A rapidly growing number of voices have argued that AI research, and computer
vision in particular, is closely tied to mass surveillance. Yet the direct path
from computer vision research to surveillance has remained obscured and
difficult to assess. This study reveals the Surveillance AI pipeline. We obtain
three decades of computer vision research papers and downstream patents (more
than 20,000 documents) and present a rich qualitative and quantitative
analysis. This analysis exposes the nature and extent of the Surveillance AI
pipeline, its institutional roots and evolution, and ongoing patterns of
obfuscation. We first perform an in-depth content analysis of computer vision
papers and downstream patents, identifying and quantifying key features and the
many, often subtly expressed, forms of surveillance that appear. On the basis
of this analysis, we present a topology of Surveillance AI that characterizes
the prevalent targeting of human data, practices of data transferal, and
institutional data use. We find stark evidence of close ties between computer
vision and surveillance. The majority (68%) of annotated computer vision papers
and patents self-report their technology enables data extraction about human
bodies and body parts and even more (90%) enable data extraction about humans
in general